%/* ----------------------------------------------------------- */
%/*                                                             */
%/*                          ___                                */
%/*                       |_| | |_/   SPEECH                    */
%/*                       | | | | \   RECOGNITION               */
%/*                       =========   SOFTWARE                  */ 
%/*                                                             */
%/*                                                             */
%/* ----------------------------------------------------------- */
%/*         Copyright: Microsoft Corporation                    */
%/*          1995-2000 Redmond, Washington USA                  */
%/*                    http://www.microsoft.com                */
%/*                                                             */
%/*   Use of this software is governed by a License Agreement   */
%/*    ** See the file License for the Conditions of Use  **    */
%/*    **     This banner notice must not be removed      **    */
%/*                                                             */
%/* ----------------------------------------------------------- */
%
% HTKBook - Steve Young 15/11/95
%

\mychap{HMM System Refinement}{Refine}

\sidepic{Tool.hedit}{55}{ } 
In chapter~\ref{c:Training}, the basic processes involved in training
a continuous density HMM system were explained and examples were given
of building a set of HMM phone models.  In the practical application
of these techniques to building real systems, there are often a number of  
problems to overcome.  Most of these arise from the conflicting desire
to have a large number of model parameters in order to
achieve high accuracy, whilst at the same time having limited and uneven
training data. \index{HMM refinement}

As mentioned previously,
the \HTK\ philosophy is to build systems incrementally.  Starting 
with a set of context-independent 
monophone HMMs, a system can be refined in a sequence of stages.  Each refinement
step typically uses the \HTK\ HMM definition editor \htool{HHEd} followed
by re-estimation using \htool{HERest}.  These incremental manipulations of
the HMM set often involve parameter tying, thus many of \htool{HHEd}'s
operations involve generating new macro definitions.

The principle types of manipulation that can be performed 
by \htool{HHEd}\index{hhed@\htool{HHEd}}
are 
\begin{itemize}
\item HMM cloning to form context-dependent model sets
\item Generalised parameter tying
\item Data driven and decision tree based clustering.
\item Mixture component splitting
\item Adding/removing state transitions
\item Stream splitting, resizing and recasting
\end{itemize}
This chapter describes how the \HTK\ tool \htool{HHEd} is used,  its
editing language and the main operations that can be
performed.

\mysect{Using \htool{HHEd}}{usingHHEd}

The HMM editor \htool{HHEd} takes as input a set of HMM definitions and
outputs a new modified set, usually to a new directory.  It is invoked
by a command line of the form
\begin{verbatim}
    HHEd -H MMF1 -H MMF2 ... -M newdir cmds.hed hmmlist
\end{verbatim}
where \texttt{cmds.hed} is an edit script containing 
a list of edit commands.  Each command
is written on a separate line and begins with a 2 letter command name.
\index{model training!HMM editing}

The effect
of  executing the above command line would be to read in the HMMs listed in 
\texttt{hmmlist} and defined
by files \texttt{MMF1}, \texttt{MMF2}, etc., apply  the editing operations
defined in \texttt{cmds.hed} and then write the resulting system out
to the directory \texttt{newdir}.  As with all tools, \HTK\ will
attempt to replicate the file structure of the input in the output
directory.  By default, any new macros generated by \htool{HHEd} will be
written to one or more of the existing MMFs.  In doing this, \HTK\ will
attempt to ensure that the ``definition before use'' rule for macros is
preserved, but it cannot always guarantee this.  Hence, it is usually
best to define explicit target file names for new macros.  This can be
done in two ways.  Firstly,   explicit target file names can be 
given in the edit script
using the \texttt{UF}\index{uf@\texttt{UF} command} command.
For example, if \texttt{cmds.hed} contained
\begin{verbatim}
   ....
   UF smacs
   # commands to generate state macros
   ....
   UF vmacs
   # commands to generate variance macros
   ....
\end{verbatim}
then the output directory would contain an MMF called \texttt{smacs} 
containing a set of state macro definitions and an MMF called \texttt{vmacs} 
containing a set of variance macro definitions, these would be in addition
to the existing MMF files \texttt{MMF1}, \texttt{MMF2}, etc.

Alternatively, the whole HMM system can be written to a single
file using the \texttt{-w} option.  For example, 
\begin{verbatim}
    HHEd -H MMF1 -H MMF2 ... -w newMMF cmds.hed hmmlist
\end{verbatim}
would write the whole of the edited HMM set to the file \texttt{newMMF}.

\index{master macro files!input/output}
As mentioned previously, each execution of \htool{HHEd} is normally followed
by re-estimation using \htool{HERest}.  Normally, all the information
needed by \htool{HHEd} is contained in the model set itself.  However,
some clustering operations require various statistics about the
training data (see sections~\ref{s:ddclust} and \ref{s:tbclust}).  
These statistics are gathered by \htool{HERest}\index{herest@\htool{HERest}} and 
output to a \textit{stats file}, which is then read in by \htool{HHEd}.
Note, however, that the statistics file\index{statistics file} 
generated by \htool{HERest}
refers to the input model set not the re-estimated set.  Thus
for example, in the following sequence, the \htool{HHEd} edit script in
\texttt{cmds.hed} contains a command 
(see the \texttt{RO} command\index{ro@\texttt{RO} command} 
in section~\ref{s:ddclust})
which references a statistics file  (called \texttt{stats}) 
describing the HMM set defined by \texttt{hmm1/MMF}.
\begin{verbatim}
    HERest -H hmm1/MMF -M hmmx -s stats hmmlist train1 train2 ....
    HHEd -H hmm1/MMF -M hmm2 cmds.hed hmmlist
\end{verbatim}
The required statistics file is generated by \htool{HERest}  but the re-estimated
model set stored in \texttt{hmmx/MMF} is  ignored and can be deleted.

\mysect{Constructing Context-Dependent Models}{mkCDHMMs}

\index{model training!context dependency}
The first stage of model refinement is usually to convert a set of
initialised and trained context-independent monophone HMMs to a 
set of context dependent models\index{context dependent models}.  As
explained in section~\ref{s:edlab}, \HTK\ uses the convention that a HMM
name of the form \texttt{l-p+r} denotes the context-dependent version of the
phone \texttt{p} which is to be used when the left neighbour is the phone
\texttt{l} and the right neighbour is the phone \texttt{r}.  To make a set
of context dependent phone models, it is only necessary to construct a HMM
list, called say \texttt{cdlist}, containing the required context-dependent models  and
then execute \htool{HHEd} with a single command in its edit script
\begin{verbatim}
    CL cdlist
\end{verbatim}
The effect of this command is that for each model \texttt{l-p+r} in \texttt{cdlist}
it makes a copy of the monophone \texttt{p}.  
\index{cloning}\index{cl@\texttt{CL} command}

The set of context-dependent models output by the above must be reestimated using
\htool{HERest}.  To do this, the training data transcriptions must be converted
to use context-dependent labels and the original monophone hmm list must be
replaced by \texttt{cdlist}.  In fact, it is best to do this conversion before
cloning the monophones because if the \htool{HLEd} \texttt{TC} 
command\index{tc@\texttt{TC} command} 
is used then the \texttt{-n} option
can be used to generate the required list of context dependent HMMs automatically. 

Before building a set of context-dependent models,
it is necessary to decide whether or not cross-word 
triphones\index{cross-word triphones} are
to be used.  If they are, then word boundaries in the training data can be ignored and
all monophone labels can be converted to triphones.  If, however, word internal triphones
are to be used, then word boundaries in the training transcriptions must be marked in
some way (either by an explicit marker which is subsequently deleted or by using a short
pause \textit{tee-model}).  This word boundary marker is then identified to \htool{HLEd}
using the \texttt{WB} command\index{wb@\texttt{WB} command} to make the \texttt{TC} command use biphones rather
than triphones at word boundaries\index{marking word boundaries} (see
section~\ref{s:edlab}).

All \HTK\ tools can read and write
HMM definitions in text or binary form.  Text is good for seeing exactly
what the tools are producing, but binary is much faster to load and store, and
much more compact.  Binary output is enabled either using the standard option
\texttt{-B} or by setting the configuration variable 
\texttt{SAVEBINARY}\index{savebinary@\texttt{SAVEBINARY}}.
In the above example, the HMM set input to \htool{HHEd} will contain a small
set of monophones whereas the output will be a large set of triphones.
In order, to save storage and computation, this is usually a good point to
switch to binary storage\index{binary storage} of MMFs. 

\mysect{Parameter Tying and Item Lists}{parmtying}

As explained in Chapter~\ref{c:HMMDefs}, \HTK\ uses macros to support a 
generalised parameter tying facility.  Referring again to
Fig.~7.\ref{f:hierarch}, each of the solid black circles denotes a potential
{\em tie-point} in the hierarchy of HMM parameters.  When two or more
parameter sets are tied, the same set of parameter values are shared by all
the {\em owners} of the tied set.  Externally, tied parameters\index{tied parameters} are
represented by macros and internally they are represented by structure 
sharing.  The accumulators needed for  the numerators and denominators of
the Baum-Welch re-estimation formulae  given in section~\ref{s:bwformulae}
are attached directly to  the parameters themselves. Hence,  when the values
of a tied parameter set are re-estimated, all of the data which would have
been used to  estimate each individual untied parameter are effectively
pooled leading to more robust parameter estimation.\index{model training!tying}

Note also that although parameter tying is implemented in
a way which makes it transparent to the \HTK\ re-estimation and recognition
tools, in practice, these tools do notice when a system has been tied
and try to take advantage of it by avoiding redundant computations.  

Although macro definitions could be written by hand, in practice,
tying is performed by executing \htool{HHEd} commands and the
resulting
macros are thus generated automatically.  The basic \htool{HHEd} command for
tying a set of parameters is the \texttt{TI} command which has the form
\begin{verbatim}
   TI macroname itemlist
\end{verbatim}
This causes all items in the given \texttt{itemlist} to be tied together
and output as a macro called \texttt{macroname}.  Macro names are
written as a string of
characters optionally enclosed in double quotes.  The latter are necessary
if the name contains one or more characters which are not letters or digits.

\sidefig{itemtree}{62}{Item List Construction}{2}{
Item lists use a simple language to identify sets of points in the 
HMM parameter hierarchy illustrated in Fig.~7.\ref{f:hierarch}.  
This language is defined fully in the reference entry
for \htool{HHEd}.
The essential idea is that item lists\index{item lists}  represent paths down the hierarchical
parameter tree where the direction {\it down} should be regarded as 
travelling from the {\it root}
of the tree to towards the {\it leaves}.  
A path can be unique, or more usually, it can
be a pattern representing a set of paths down the tree.  The point at
which each path stops identifies one member of the set represented by
the item list.  
Fig.~\href{f:itemtree} shows the possible paths down the tree.  In
text form the branches are replaced by dots and the underlined node
names are possible terminating points.  At the topmost level, an
item list is a comma separated list of paths enclosed in braces.
}

Some examples, should make all this clearer.  Firstly, the
following is a legal but somewhat long-winded way of specifying
the set of items comprising states 2, 3 and 4 of the HMM called \texttt{aa}
\begin{verbatim}
     { aa.state[2],aa.state[3],aa.state[4] }
\end{verbatim}
however in practice this would be written much more compactly as
\begin{verbatim}
     { aa.state[2-4] }
\end{verbatim}
It must be emphasised that indices in item lists are really {\it patterns}.
The set represented by an item list consists of all those elements which
match the patterns.  Thus, if \texttt{aa} only had two emitting states, the above item
list would not generate an error.  It would simply only match two items.
The reason for this is that the same pattern can be applied to many different
objects.  For example, the HMM name can be replaced by a list of names 
enclosed in brackets, furthermore each HMM name can include `?' characters
which match any single character and `*' characters which match zero or
more characters.  Thus \index{item lists!pattern matching}
\begin{verbatim}
     { (aa+*,iy+*,eh+*).state[2-4] }
\end{verbatim}
represents states 2, 3 and 4  
of all biphone models corresponding to
the phonemes \texttt{aa}, \texttt{iy} and \texttt{eh}.  If \texttt{aa} had just 2 emitting
states and the others had 4 emitting states, then this item list would include
2 states from each of the \texttt{aa} models and 3 states from
each of the others.  Moving further down the tree, the item list
\begin{verbatim}
     { *.state[2-4].stream[1].mix[1,3].cov }
\end{verbatim}
denotes the set of all covariance vectors (or matrices) of the first and
third mixture
components of stream 1, of states 2 to 4 of all HMMs.  Since many HMM systems
are single stream, the \texttt{stream} part of the path can be omitted if its value
is 1.  Thus, the above could have been written
\begin{verbatim}
     { *.state[2-4].mix[1,3].cov }
\end{verbatim}
These last two examples also show that indices\index{item lists!indexing} can be written as comma
separated lists as well as ranges, for example, \texttt{[1,3,4-6,9]}
is a valid index list representing states 1, 3, 4, 5, 6, and 9.

When item lists are used as the argument to a \texttt{TI} 
command\index{ti@\texttt{TI} command}, the
kind of items represented by the list determines the macro type in a fairly
obvious way.  The only non-obvious cases are firstly that lists ending
in \texttt{cov} generate \hmmt{v}, \hmmt{i}, \hmmt{c}, or \hmmt{x} macros as
appropriate.   If an explicit set of mixture components is defined
as in
\begin{verbatim}
     { *.state[2].mix[1-5] }
\end{verbatim}
then  \hmmt{m} macros are generated but omitting
the indices  altogether denotes a special case of mixture 
tying\index{tied-mixtures}
which is explained later in Chapter~\ref{c:discmods}.

To illustrate the use of item lists, some example \texttt{TI} commands
can now be given.  Firstly, when a set of context-dependent models is created, it can
be beneficial to share one transition matrix across all variants
of a phone rather than having a distinct transition matrix for each.
This could be achieved by adding \texttt{TI}
commands immediately after the \texttt{CL} command described in
the previous section, that is\index{tying!examples of}
\begin{verbatim}
    CL cdlist
    TI T_ah {*-ah+*.transP}
    TI T_eh {*-eh+*.transP}
    TI T_ae {*-ae+*.transP}
    TI T_ih {*-ih+*.transP}
     ... etc
\end{verbatim}

As a second example, a so-called Grand Variance\index{grand variance} 
HMM system can
be generated very easily with the following HHEd command
\begin{verbatim}
     TI "gvar" { *.state[2-4].mix[1].cov }
\end{verbatim}
where it is assumed that the HMMs are 3-state 
single mixture component models.   The effect
of this command is to tie all state distributions to a single global variance
vector.  For applications, where there is limited training data, this technique
can improve performance, particularly in noise.

Speech recognition systems will often have distinct
models for silence  and short pauses.  A 
silence model\index{silence model} \texttt{sil} may have
the normal 3 state topology whereas a short pause model may have just 
a single state.  To avoid the two models \textit{competing} with each other, the
\texttt{sp} model state can be tied to the centre state of the \texttt{sil} model
thus
\begin{verbatim}
     TI "silst" { sp.state[2], sil.state[3] }
\end{verbatim}

So far nothing has been said about how the parameters are actually
determined when a set of items is replaced by a single shared representative.
When states are tied, the state with the broadest  variances  and as few as
possible zero mixture component weights is selected from the pool and used
as the representative.  When mean vectors are tied, the average of all the
mean vectors in the pool is used and when variances are tied, the largest
variance in the the pool is used.  In all other cases, the last item in the
tie-list is  arbitrarily chosen as representative.
All of these selection criteria are \textit{ad hoc}, but since
the tie operations are always followed by explicit re-estimation
using \htool{HERest}, the precise choice of representative for a tied
set is not critical.\index{tying!exemplar selection}

Finally, tied parameters can be
untied.  For example,  subsequent refinements of the context-dependent model set
generated above with tied transition matrices might result in
a much more compact set of models for which individual transition
parameters could be robustly estimated.    This 
can be done using the \texttt{UT} command\index{ut@\texttt{UT} command} whose effect is to untie all of the
items in its argument list.  For example, the command
\begin{verbatim}
     UT {*-iy+*.transP}
\end{verbatim}
would untie the transition parameters in all variants of the \texttt{iy}
phoneme.
This untying works by simply making unique copies of the tied parameters.
These untied parameters can then subsequently be re-estimated.

\mysect{Data-Driven Clustering}{ddclust}
 
In
section~\ref{s:mkCDHMMs}, a method of triphone construction was described
which involved cloning all monophones and then re-estimating them using data
for which monophone labels have been replaced by triphone labels.  
This will lead to a very large set of models, and relatively little
training data for each model.  Applying the argument that context will not greatly affect
the centre states of triphone models, one way to reduce the 
total number of parameters without 
significantly altering the models' ability to represent the different
contextual effects might be to tie all of the centre states across all
models derived from the same monophone.  This tying could 
be\index{clustering!data-driven}
done by writing an edit script of the form
\begin{verbatim}
     TI "iyS3" {*-iy+*.state[3]}
     TI "ihS3" {*-ih+*.state[3]}
     TI "ehS3" {*-eh+*.state[3]}
      .... etc
\end{verbatim}
Each \texttt{TI} command would tie all the centre states of all triphones
in each phone group. Hence, if there were an average of 100 triphones
per phone group then the total number of states per group
would be reduced from
300 to 201.

Explicit tyings such as these can have some positive effect but overall they 
are not very satisfactory.  Tying all centre states is too severe and worse
still, the problem of undertraining for the left and right states remains.
A much better approach is to use clustering to decide which states to
tie.  \htool{HHEd} provides two mechanisms for this.  In this section
a data-driven clustering approach will be described and in
the next section, an alternative decision tree-based approach is presented.

Data-driven clustering is performed by the \index{model training!clustering}
\texttt{TC}\index{tc@\texttt{TC} command} and 
\texttt{NC}\index{nc@\texttt{NC} command}
commands.  These both invoke the same top-down hierarchical
procedure.  Initially all states are placed in individual
clusters.  The pair of clusters which when combined would form the smallest
resultant cluster are merged.  This process repeats until either the
size of the largest
cluster reaches the threshold set by the \texttt{TC} command or
the total number of clusters has fallen to that
specified by by the \texttt{NC} command.  The size of cluster
is defined as the greatest distance between any two states.
The distance metric depends on the type of state distribution.
For single Gaussians, a weighted Euclidean distance between the means
is used and for tied-mixture systems a  Euclidean distance between the
mixture weights is used.  For all other cases, the average probability
of each component mean with respect to the other state is used.
The details of the algorithm and these metrics are given in the reference
section for \htool{HHEd}.

\centrefig{tiedstate}{100}{Data-driven state tying}

As an example, the following \htool{HHEd} script would cluster and tie the
corresponding states of the triphone group for the phone \texttt{ih}
\begin{verbatim}
     TC 100.0 "ihS2" {*-ih+*.state[2]}
     TC 100.0 "ihS3" {*-ih+*.state[3]}
     TC 100.0 "ihS4" {*-ih+*.state[4]}
\end{verbatim}
In this example, each \texttt{TC} command performs clustering on the specified
set of states, each cluster is  tied and output as a macro.  The macro name
is generated by appending the cluster index to
the macro  name given in the command.   The effect of this command 
is illustrated in Fig.~\href{f:tiedstate}.  Note that if a word-internal
triphone system is being built, it is sensible to include biphones as well
as triphones in the item list, for example, the first command above would
be written as
\begin{verbatim}
     TC 100.0 "ihS2" {(*-ih,ih+*,*-ih+*).state[2]}
\end{verbatim}
If the above \texttt{TC} commands are repeated for all phones, the resulting  set of
tied-state models will have far
fewer parameters in total than the original untied set.  The numeric argument
immediately following the \texttt{TC} command name is the cluster threshold.  Increasing
this value will allow larger and hence, fewer clusters. The aim, of
course, is to strike the right balance between compactness and the acoustic
accuracy of the individual models.  In practice, the use of this command
requires some experimentation to find a good threshold value. \htool{HHEd} provides
extensive trace  output for monitoring clustering operations.  Note in this
respect that as well as setting tracing from the command line and the
configuration file, tracing in \htool{HHEd} can be set by the \texttt{TR} command. 
Thus,  tracing can be controlled at the command level. Further trace
information can be obtained by including the \texttt{SH} command\index{sh@\texttt{SH} command} at strategic
points in the edit script.  The effect of executing this command is to list
out all of the parameter tyings currently in force.

A potential problem with the use of the \texttt{TC} and \texttt{NC} commands is
that {\it outlier} states will tend to form their own singleton 
clusters\index{singleton clusters} for
which there is then insufficient data to properly train.  One solution to
this is to use the \texttt{RO} command\index{ro@\texttt{RO} command} to 
remove outliers\index{removing outliers}.  This command has
the form
\begin{verbatim}
     RO thresh "statsfile"
\end{verbatim}
where \texttt{statsfile} is the name of a statistics file\index{statistics
file} output using the
\texttt{-s} option of \htool{HERest}.  This statistics file holds the 
{\em occupation counts} for all states of the HMM set being trained.  
The term {\em occupation count} refers to the number of frames allocated to a
particular state and can be used as a measure of how much training data is
available for estimating the parameters of that state.  
The \texttt{RO} command must be executed {\it before} the \texttt{TC} or
\texttt{NC} commands used to do the actual clustering. Its effect is to simply
read in the statistics information from the given file and then to set a flag
instructing the
\texttt{TC} or \texttt{NC} commands to remove any outliers remaining at 
the conclusion
of the normal clustering process.  This is done by repeatedly finding the
cluster with the smallest total occupation count and merging it with its
nearest neighbour. This process is repeated until all clusters have a total
occupation count which exceeds \texttt{thresh}, thereby ensuring that every
cluster of states will be properly trained in the subsequent re-estimation
performed by \htool{HERest}.\index{state tying}


On completion of the above clustering and tying procedures, many of the models
may be effectively identical, since acoustically similar triphones may share
common clusters for all their emitting states.  They are then, in effect,
so-called {\it generalised triphones}.\index{generalised triphones} State tying
can be further exploited if the HMMs which are effectively equivalent are
identified and then tied via the physical-logical mapping\footnote{The physical
HMM which corresponding to several logical HMMs will be arbitrarily named after
one of them.} facility provided by HMM lists (see section~\ref{s:hmmsets}). The
effect of this would be to reduce the total number of HMM definitions required.
\htool{HHEd} provides a compaction command to do all of this automatically.
For example, the command
\begin{verbatim}
     CO newList 
\end{verbatim}
\index{co@\texttt{CO} command}will compact\index{model training!compacting} 
the currently loaded HMM set by identifying equivalent models
and then tying them via the new HMM list output  to the 
file \texttt{newList}.  Note, however, that for two HMMs to be tied, they
must be identical in all respects.
This is one of the reasons why transition parameters are often tied
across triphone groups otherwise HMMs with identical states would still
be left distinct due to minor differences in their transition matrices.

\mysect{Tree-Based Clustering}{tbclust}

\index{clustering!tree-based}
One limitation of the data-driven clustering procedure described above is
that it does not deal with triphones for which there are no examples in the
training data.  When building word-internal triphone systems,  this 
problem can often
be avoided by careful design of the training database but when building large
vocabulary cross-word triphone systems \textit{unseen} triphones are unavoidable.
\index{unseen triphones}

\centrefig{qstree}{100}{Decision tree-based state tying}

\htool{HHEd} provides an alternative decision 
tree based clustering\index{decision tree-based clustering} mechanism
which provides a similar quality of clustering but offers a solution 
to the unseen triphone problem.  Decision tree-based clustering is invoked
by the command \texttt{TB} which is analogous to the \texttt{TC} command
described above and has an identical form, that is
\begin{verbatim}
    TB thresh macroname itemlist
\end{verbatim}
Apart from the clustering mechanism, there are some other differences between
\texttt{TC} and \texttt{TB}.  Firstly, \texttt{TC} uses a distance metric between
states whereas \texttt{TB} uses a log likelihood criterion.  Thus, the threshold
values are not directly comparable.  Furthermore, \texttt{TC} supports any type
of output distribution whereas \texttt{TB} only supports single-Gaussian
continuous density output distributions.
Secondly, although the following describes only state clustering, 
the \texttt{TB} command\index{tb@\texttt{TB} command} can also be used to cluster whole
models.

A phonetic decision tree is a binary tree in which a yes/no phonetic 
question\index{phonetic questions}
is attached to each node.  
Initially all states in a given item list (typically a specific phone state position)
are placed at the
root node of a tree. Depending on each answer, the pool of states is
successively split and this continues until the states have trickled
down to leaf-nodes.  All states in the same leaf node are then tied. 
For example, Fig~\href{f:qstree} illustrates the case of tying the centre
states of all triphones of the phone /aw/ (as in ``out'').  All of the states trickle
down the tree and depending on the answer to the questions, they end up
at one of the shaded terminal nodes.   For example, in the illustrated
case, the centre state of \texttt{s-aw+n} would join the second leaf
node from the right since its right context is a central consonant,
and its right
context is a nasal  but its left context is not a central stop.

The question at each node is chosen to (locally) maximise the likelihood
of the training data given the final set of state tyings. 
Before any tree building can take place, all of the possible phonetic
questions must be loaded into \htool{HHEd} using \texttt{QS} 
commands\index{qs@\texttt{QS} command}.  Each
question takes the form ``Is the left or right context in the set P?'' where the
context is the model context as defined by its logical name.  The 
set P is
represented by an item list and
for convenience every question is given a name.  As an example, the
following command 
\begin{verbatim}
    QS "L_Nasal" { ng-*,n-*,m-* }
\end{verbatim}
defines the question ``Is the left context a nasal?''.

It is possible to calculate the log likelihood of the training
data given any pool of states (or models).  Furthermore, this 
can be done without reference to the
training data itself since for single Gaussian distributions the means, variances
and state occupation counts (input via a stats file) form sufficient statistics.
Splitting any pool into two will increase the log likelihood since it provides twice
as many parameters to model the same amount of data.  The increase obtained when
each possible question is used can thus be calculated and the question selected
which gives the biggest improvement.  

Trees are therefore built using a top-down sequential optimisation process.
Initially all states (or models) are placed in a single cluster at the root
of the tree.  The question is then found which gives the best split of the root
node.  This process is repeated until the increase in log likelihood falls
below the threshold specified in the \texttt{TB} command.
As a final stage, the decrease in log likelihood is calculated for merging
terminal nodes with differing parents.  Any pair of nodes for which this
decrease is less than the threshold used to stop splitting are then merged.
\index{tree optimisation}

As with the \texttt{TC} command, it is useful to prevent the creation of
clusters with very little associated training data.  The \texttt{RO} command
can therefore be used in tree clustering as well as in data-driven clustering.
When used with trees, any split which would result in a total occupation count
falling below the value specified is prohibited.  Note that the \texttt{RO}
command can also be used to load the required stats file.  Alternatively,
the stats file can be loaded using the \texttt{LS} command\index{ls@\texttt{LS} command}.

As with data-driven clustering, using the trace facilities provided by
\htool{HHEd} is recommended for monitoring and setting the appropriate thresholds.
Basic tracing provides the following summary data for each tree
\begin{verbatim}
    TB 350.00 aw_s3 {}
     Tree based clustering
      Start  aw[3] : 28  have  LogL=-86.899 occ=864.2
      Via    aw[3] : 5   gives LogL=-84.421 occ=864.2
      End    aw[3] : 5   gives LogL=-84.421 occ=864.2
    TB: Stats 28->5 [17.9%]  { 4537->285 [6.3%] total }
\end{verbatim}
This example corresponds to the case illustrated in Fig~\href{f:qstree}.
The \texttt{TB}
command has been invoked with a  threshold of 350.0 to cluster
the centre states of the triphones of the phone \textit{aw}.
At the start of clustering with all 28 states in a single pool, the average
log likelihood per unit of occupation is -86.9 and on completion with
5 clusters this has increased to -84.4.  The middle line labelled ``via'' gives
the position after the tree has been built but before terminal nodes have been
merged (none were merged in this case).  The last line summarises the overall
position.  After building this tree, a total of 4537 states were reduced
to 285 clusters.\index{clustering!tracing in}

As noted at the start of this section, an important advantage of tree-based clustering
is that it allows triphone models which have no training data to be synthesised.
This is done in \htool{HHEd} using the \texttt{AU} command\index{au@\texttt{AU} command} which has the form
\begin{verbatim}
    AU hmmlist
\end{verbatim}
Its effect is to scan the given \texttt{hmmlist} and any physical models listed
which are not in the currently loaded set are synthesised.  This is done 
by descending the previously constructed trees for that phone and answering the
questions at each node based on the new unseen context.  When each leaf node is
reached, the state representing that cluster is used for the corresponding state
in the unseen triphone\index{unseen triphones!synthesising}.

The \texttt{AU} command can be used within the same edit script as the tree building
commands.  However, it will often be the case that a new set of triphones is needed
at a later date, perhaps as a result of vocabulary changes.  To make this possible,
a complete set of trees can be saved using the \texttt{ST} 
command\index{st@\texttt{ST} command} and then later
reloaded using the \texttt{LT} command\index{lt@\texttt{LT} command}.
\index{decision trees!loading and storing}

\mysect{Mixture Incrementing}{upmix}

When building sub-word based continuous density systems, 
the final system will typically consist of multiple mixture component
context-dependent HMMs.  However, as indicated previously, the early
stages of triphone construction, particularly state tying, are best done
with single Gaussian models.  Indeed, if tree-based clustering is to be
used there is no option.\index{mixture incrementing}\index{up-mixing}

In \HTK\ therefore, the conversion from single Gaussian HMMs to multiple
mixture component HMMs is usually one of the final steps in building
a system.  The mechanism provided to do this is the \htool{HHEd} \texttt{MU} command
which will increase the number of components in a mixture by 
a process called \textit{mixture splitting}.
This approach to building a multiple
mixture component system is extremely
flexible since it allows the number of mixture components to be repeatedly increased
until the desired level of performance is achieved.\index{mixture splitting}

The \texttt{MU} command\index{mu@\texttt{MU} command} has the form
\begin{verbatim}
     MU n itemList
\end{verbatim}
where \texttt{n} gives the new number of mixture components required
and \texttt{itemList} defines the actual mixture distributions to
modify.  This command works by repeatedly splitting the 'heaviest'
mixture component until the required number of components is obtained.
The 'heaviness' score of a mixture component is defined as the mixture
weight minus the number of splits involving that component that have
already been carried out by the current MU command. Subtracting the
number of splits discourages repeated splitting of the same mixture
component. If the GCONST value~\ref{s:hmmdef} of a component is more
than four standard deviations smaller than the average gConst, a
further adjustment is made to the 'heaviness' score of the component
in order to make it very unlikely that the component will be selected
for splitting. The actual split is performed by copying the mixture
component, dividing the weights of both copies by 2, and finally
perturbing the means by plus or minus 0.2 standard deviations.  For
example, the command
\begin{verbatim}
     MU 3 {aa.state[2].mix}
\end{verbatim}
would increase the number of mixture components in the output distribution
for state 2 of model \texttt{aa} to 3.   Normally, however,  the number of
components in all mixture distributions will be increased at the same time.
Hence, a command of the form is more usual
\begin{verbatim}
     MU 3 {*.state[2-4].mix}
\end{verbatim}
It is usually a good idea to increment mixture components in
stages, for example, by incrementing by 1 or 2 then re-estimating, then
incrementing by 1 or 2 again and re-estimating, and so on until the
required number of components are obtained.  This also allows recognition performance
to be monitored to find the optimum.

One final point with regard to multiple mixture component distributions is that
all \HTK\ tools ignore mixture components whose weights fall below a threshold value
called \texttt{MINMIX} (defined in \texttt{HModel.h}).  Such mixture components
are called {\it defunct}.  Defunct mixture components can be
prevented by setting the \texttt{-w} option in \htool{HERest} so that all mixture
weights are floored to some level above 
\texttt{MINMIX}\index{minmix@\texttt{MINMIX}}.  If 
mixture weights\index{mixture weight floor}
are allowed to fall below \texttt{MINMIX} then the corresponding Gaussian
parameters will not be written out when the model containing that component
is saved.  It is possible to recover from this, however, since the \texttt{MU} command
will replace defunct mixtures\index{defunct mixtures} before performing any requested mixture
component increment.

\mysect{Regression Class Tree Construction}{hhedregtree}

In order to perform most model adaptation tasks (see
chapter~\ref{c:Adapt}), it will be necessarily to produce a binary regression
class tree\index{adaptation!regression tree}. 
This tree is stored in the MMF, along with a regression
base class identifier for each mixture component. An example
regression tree and how it may be used is shown in
subsection~\ref{s:reg_classes}. \htool{HHEd} provides the means to
construct a regression class tree for a given MMF, and is invoked
using the \texttt{RC} command\index{rc@\texttt{RC} command}. 
It is also necessary to supply a
statistics file, which is output using the \texttt{-s} option of
\texttt{HERest}. The statistics file can be loaded by invoking the
\texttt{LS}\index{ls@\texttt{LS} command} command.

A centroid-splitting algorithm using a Euclidean distance measure is
used to grow the binary regression class tree to cluster the model
set's  mixture components.  Each leaf node therefore specifies a 
particular mixture component cluster. This algorithm proceeds
as follows until the requested number of terminals has been achieved.
\begin{itemize}
\item Select a terminal node that is to be split.
\item Calculated the mean and variance from the mixture components clustered
at this node.
\item Create two children. Initialise their means to the parent mean 
perturbed in opposite directions (for each child) by a fraction of 
the variance.
\item For each component at the parent node assign the component 
to one of the children by using a Euclidean distance measure to 
ascertain which child mean the component is closest to.
\item Once all the components have been assigned, calculate the new
means for the children, based on the component assignments.
\item Keep re-assigning components to the children and re-estimating
the child means until there is no change in assignments from one
iteration to the next. Now finalise the split.
\end{itemize}

As an example, the following \htool{HHEd} script would produce a 
regression class tree with 32 terminal nodes, or regression base
classes:-
\begin{verbatim}
     LS "statsfile"
     RC 32 "rtree"
\end{verbatim}

A further optional argument is possible with the 
\texttt{RC} command. 
This argument allows the user to specify the non-speech class
mixture components using an \texttt{itemlist}, 
such as the silence mixture components. 
\begin{verbatim}
     LS "statsfile"
     RC 32 "rtree" {sil.state[2-4].mix}
\end{verbatim}
In this case the first split that will be made in the regression class tree
will be to split the speech and non-speech sounds, after which the
tree building continues as usual. 

\mysect{Miscellaneous Operations}{misedit}

The preceding sections have described the main \htool{HHEd} commands used for
building continuous density systems with tied parameters.  A further group
of commands (\texttt{JO}, \texttt{TI} and \texttt{HK}) are used to build
tied-mixture systems and these are described in Chapter~\ref{c:discmods}.
Those
remaining cover a miscellany of functions.  They are documented in the
reference entry for \htool{HHEd} and include commands to add and remove
state transitions\index{state transitions!adding/removing} 
(\texttt{AT}\index{at@\texttt{AT} command},
\texttt{RT}\index{rt@\texttt{RT} command}); synthesise triphones from
biphones (\texttt{MT}\index{mt@\texttt{MT} command}); 
change the parameter kind of a HMM (\texttt{SK}\index{sk@\texttt{SK} command});
modify stream dimensions (\texttt{SS}\index{ss@\texttt{SS} command},
\texttt{SU}\index{su@\texttt{SU} command},
\texttt{SW}\index{sw@\texttt{SW} command}); change/add an identifier name
to an MMF (\texttt{RN}\index{rn@\texttt{RN}} command); and expand
HMM sets by duplication, for example, as needed in making gender
dependent models (\texttt{DP}\index{dp@\texttt{DP} command}).


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "htkbook"
%%% End: 
